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(57) A method and system for employing image rec- 
ognition techniques to produce a photocoilage from a 
plurality of images wherein the system obtains a digital 
record for each of the plurality of images, assigns each 
of the digital records a unique identifier and stories the 
digital records in a database; automatically sorts the 
digital records using ait least one date type to categorize 
each of the digital records according at least one prede- 
termined criteria; employs means responsive to the 
sorting step to compose a photocoilage from the digital 
records. The method and system employ, data tyjpes 
selected from pixel data; metadata; product order infor- 
mation; processing goal information; or customer profile 
to automatically sort data typically by culling or grouping 
to categorize according to either an event, a person, or 
chronologically 
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Description 

FIELD OF THE INVENTION 

[0001] The invention relates generally to the field of 
photography, and in particular to photo collections. More 
spectfically, the invention relates to employing image 
recognition techniques tor generating j3hotocollages 
automatically. 

BACKGROUND OF THE INVENTION 

[0002] Photographs, videos, and memorabilia col- 
lections are very commonly used to maintain memories 
and events that formed a part of a persons life. These 
collections serve to augment the human memory and 
enrich the process of sharing stories related to the 
memories. When organized, viewed and shared on a 
regular basis a collection of memory artifacts generates 
a large reward, enriching the lives of all involved. The 
nature of these collections is such that they grow stead- 
ily, event by event, year by year, and soon become large 
and difficult to manage. Collections of photos and menrv 
orabilia are considered one of the most important and 
valued possessions by most people. They are the first 
things that people think of when forced to flee their 
homes due to fire, flood or other natural disaster. These 
collections possess intrinsic, emotional value, even if 
they are never viewed, because the need to preserve a 
memory of life is strong and universal. Because of the 
relative importance of these memories to the persons 
involved, the prior art is replete with teachings that dis- 
close organizational methods. 

[0003] The most common manner of organizing 
these collections within the prior art is to place the pho- 
tos, videos or memorabilia into either an album or a box. 
Common vinyl album pages provide the means to store 
and view between one and five standard sized photos 
per page. Creative people often spend hours carefully 
selecting and arranging photos, writing captions, clip- 
ping newspaper articles, and other memorabilia to cre- 
ate visual stories or scrapbooks. Once organized into 
groups or pages these photocollages greatly enhance a 
person's ability to remember and share the story sur- 
rounding the depicted events. These simple organiza- 
tion tools allow the collections to be easily viewed and 
also serves to protect the artifacts themselves. There 
are numerous types of albums and boxes available in 
the market today, ranging from simple vinyl sleeves to 
boxes manufactured from specialized materials 
designed to preserve the artifacts. Album vendors 
include Pioneer Photo Albums, Design Vinyl and 
Cason-Talens. Box vendors include Exposures. None of 
these prior art disclosures provide a means by which a 
photocollage of these memorable events can be easily 
constructed by persons to who these event means so 
much. 

[0004] As used herein photocollage refers to a sin- 



gle page having a plurality of images, such as a page in 
a photo album, or a composite image having a number 
of images relating to a single theme such as a vacation, 
wedding, birthday party or the like. The concept of pho- 

5 tocollage as used herein also includes the concept of a 
bound photo album having a plurality of pages, one or 
more of which is a photocollage. Despite the fact that 
many people are engaged in collecting these memora- 
ble artifacts, few people have the free time available to 

10 invest on a regular t>asis to organize and maintain them. 
Before long, the amount of unorganized material 
becomes a significant psychological barrier to getting 
organized. Other barriers exist which prevent people 
from actively maintaining these memorab'lia collections 

IS such as confidence in their process, access to the mate- 
rials, or remembering the details about the event Often, 
once people get started on this organizattonal task they 
find it rewarding and fun, but still a significant amount of 
work. 

20 [0005] Many attempts have been made to provide 
tools for working with or organizing photo and memora- 
bilia collections. Computer software programs such as 
Picture-It™, by Microsoft, or Creative Photo Albums™, 
by Dog Byte Devetopment, allow people to work with 

25 digital versions of their photos and aeate digital ver- 
sions of an album or print them on a home printer. Soft- 
ware products such as these require each photo or 
artifact exist in digital form before they can be used. 
Although these products increase the ability to change 

30 and enhance photos and scanned memorabilia they do 
not reduce the amount of work needed to organize col- 
lections or create visual stories. Other services such as 
Photo-Net™ by PictureVision™ will scan photographs in 
a high-quality format at the time of photo processing 

35 and provide a thumbnail image of the scanned images 
via the Internet A customer, using these scanned 
images can create collections of photos which can be 
viewed on the Internet or have prints generated. Cur- 
rently some of these services do not allow for the 

40 arrangement of several photos on a page and are lim- 
ited to consumers who have a collection of digital 
images and a computer connected to the Internet and 
who are both computer and web literate. 
[0006] It should be apparent from the foregoing cfis- 

45 cussion that there remains a need within the art for a 
method by which consumers can create photocollages 
and photo albums (or have them made for them) in a 
manner that is as simple as ordering prints. 



[0007] The present invention addresses the need 
for an improved method of generating photo albums 
from consumer photographs that requires a minimum 
55 amount of effort but yields a high-quality product and is 
reasonably priced. 

[0008] The present invention is directed to over- 
coming one or more of the problems set forth above. 
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Briefly summarized, according to one aspect of the 
present invention, a system and method for producing a 
photocollage from a plurality of images, comprising the 
steps of: a) obtaining a digital record for each of the plu- 
rality of images, each of the digital records having a 
unique identifier and storing the dgital records in a 
database; b) automatically sorting the digital records 
using at least one date type to categorize each of the 
digital records according at least one predetermined cri- 
teria; c) enploying means responsive to the sorting step 
to compose a photocollage from the digital records. The 
system then associates each of the images with at least 
one of the categories followed by a sorting step that 
arranges the images according to the categories. The 
system then employs the categories to automatically 
construct the photocollage from tiie stored images by 
generating a plurality of pages of the stored images. 
[0009] These and other aspects, objects, features 
and advantages of the present invention will be more 
clearly understood and appreciated froni a review of the 
following detailed description of tiie preferred embodi- 
m^ts and appended claims, and by reference to the 
accompanying drawings. 

ADVANTAGEOUS EFFECT OF THE INVENTION 

[0010] The present invention has the following 

advantages: Allows the user to have (1) an easy metfiod 
for creating professional looking photocollages, (2) 
duplication of photocollages, and (3) keeping photocol- 
lage fOes for later use.. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0011] 

FIQ. 1 is a block diagram illustrating tiie basic sys- 
tem elements, used in practicing the present inven- 
tion; and 

FIG. 2 is a system diagram showing the collection 
steps tiiat take place once a customer has deliv- 
ered images to tine system; 
FIG. , 3 is a f tow diagram showing the active 
processing goal steps used by the system of the 
present invention; 

FIG. 4 is a flow chart showing the steps performed 
by the present invention; towards a story prepara- 
tion based photocollage. 

[0012] To facilitate understanding, identical referr 
ence numerals have been used, where possible, to des- 
ignate identical elements that are common to the 
figures. 

DETAILED DESCRIPTION OF THE INVENTION 

[0013] An acceptable photo album or photocollage 
can be created automatically from customers' exposed 



f nm, negatives, or digital images. 
[0014] Referring to Fig. 1, which is the method as 
envisioned by the present invention designed to auto- 
matically produce photocollages including albums 17, 

5 CDs 15 and other image-based keepsakes 11. The 
process of transforming the supplied Image n^aterial 
into a photocollage is referred to as Story Preparation 
services. Story Preparation services applies the neces- 
sary amount of image understanding embodied in a col- 

10 lection of processing modules executed in a non-linear 
sequence to create a story representation of the 
rmagea In a preferred embodiment tiiese processing 
modules include: collecting, sorting, culling, annotating, 
grouping, enhancing, associating and. composing. 

IS These steps can be performed in the order listed or they 
may be rearranged in different sequences depending 
upon the desired output product. The production of a 
particular photocollage product may include one, two or, 
more of the available steps. Which steps are used and 

20 in what sequence is determined by the system based 
upon the desired product. It is importarrt to remember 
that the present invention envisions providing sufficient 
image understanding of the subject images with the ulti- 
mate goal of teaching the system to understand that 

25 there are pervasive themes that exist wltiiin various sets 
of images and recognizing these themes thereby creat- 
ing an image story product, or photocollage 
[001 5] As shown in Rg. 2, the collection step begins 
when a customer, having completed picture taking for 

30 one or more events delivers one or more exposed film 
strips or cartridges 1 0, digital still camera rhemory cards 
12, photographic prints 6 or video camera media 8 to a 
processing facility 14. At the time that the customer 
delivers the exposed f ijm cartridge(s) to the processing 

35 fadlrty the customer's identity is recorded and assod- 
. ated with the suite of film cartridges and other image 
sources. Altematively the customer identity may be 
encoded on the film by. exposing a machine-readable 
sequence of marks along the edge of the film or by 

40 using the magnetic coating on the Advanced Photo Sys- 
tem film. The conventional film processing takes place 
with conventional correction oi exposed photographs to 
balance color and, brightness and contrast The 
exposed film images are chemically processed 16 to 

45 generate an optical image. These optical images are 
then scanned 18 to produce a hi^-resolution digital file 
that is archived in a data store 20. In general, to produce 
a high-resolution printed image of 8 x 10 inch size, a 
resolution of 1538 x 1024 pixels is required. Digital Still 

so Camera images from Digital Still Camera memory cards 
that are delivered to the processing facility are digitally 
processed 22 to produce an equivalent digital image file 
of similar size and stored on the data store 20. Analog 
video camera images on media 8 delivered to the 

55 processing fadlity are digitized and processed before 
storage on the data store 20. At the conclusion of this 
step in the process there exists a collection of digiteU 
image files assodated with a customer by means of a 
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customer identification. This coltection of digital innage 
data/ or pixel data 32. is now available as a source of 
data for the Story Sen^ices processing modules. Data 
attached or associated with individual images or groups 
of images provided by the customer such as date, time, 
location, sound, camera identification, customer identifi- 
cation, exposure parameters, Advanced Photo System 
IX data, are all examples of meta-data. This meta-data 
34 is stored on image store 20 and associated with the 
available pixel data 32. In addition the customer selects 
the desired output product or product type from a listing 
of available choices. This selection may be indicated by 
the customer marking the photo processing bag 24, ver- 
bally indicating the desired product to a clerk or by input 
at a computer terminal or kiosk 28. The order descrip- 
tion 36 generated from any one of the indicated sources 
provided by the customer is stored on the system and is 
associated, by means of a customer identification, with 
an existing customer profile 30 in the data store 20. 
[0016] At the conclusion of the collection step the 
pixel data, meta-data, customer profile and order 
description are available as input data for the subse- 
quent processing steps. These steps can be performed 
in the order listed or they may be rearranged in different 
sequences depending upon the desired output product. 
The production of a particular photocollage product may 
include one, two or, more of the available steps. Which 
steps are used and in what sequence is determined by 
the system based upon the desired product Following 
the collection step but before subsequent processing 
steps are performed, the specific processing to be 
applied to the collected set of image material is deter- 
mined in a process goal generation step 37 arxi embod- 
ied in a set of photocollage processing goals 38. For 
each of the output products requested the system deter- 
mines an optimum sequence of processing steps to be 
applied. The processing of the output product or prod- 
ucts will follow this system determined processing 
sequence, unless the sequence is modified by one of 
the processing modules. The processing output for 
each module is stated as one or more processing step 
goals. The processing in each step uses, the stated 
processing step goals to control the logic, rules, and 
parameters applied during the processing of the col- 
lected image material to satisfy the stated processing 
step goal or goals. In addition to accomplishing 
processing goals each processing step can also create 
goals for itself or future processing steps. New goals, 
which result from executing a particular processing 
step, reflect necessary adjustments of desired parame- 
ters because the original goal is inpossible to achieve 
or the processing step may generate new goals for sut>- 
sequent processing steps. The initial set of processing 
goals 38 is determined by retrieving a default process- 
ing description from a database of available products 40 
maintained by the system. Once the initial set of 
processing goals is determined the photocollage 
processing is initiated by the system. 



[0017] As described, the automatic processing of 
photocollages is carried out in processing nx>dules 
employed to perform these image understanding and 
interpretation steps. Each processing module relies on 

5 data from a number of sources in order to satisfy the 
processing goal. Each processing module has access 
to five sources of data: pixel information contained in the 
individual images, meta-data attached to images or 
groups of images, the original product or service order 

10 description, the list of active processing goals main- 
tained by the system and, a customer profile containing 
information about the customer. The pixel data 32 for 
each image provides each module witii the opportunity 
to apply image processing and image understanding 

15 algorithms. The meta-data 34 as used herein refers to 
the information attached to the individual images and to 
groups of images will contain infbrmatbn about the 
image or group of images which originated at the time of 
capture or was generated in prior processing of the 

20 image or group of images. Date, time, location, sound, 
camera identification, customer identification, exposure 
parameters. Advanced Photo System IX data, are all 
examples of meta-data that are, in a preferred emt>odi- 
ment be attached to the original input images. The orig- 

25 inal product or service order description 36 contains the 
specific product request from the customer and any 
notes that were captured in the ordering process. At any 
time in the processing of the image or group of images 
the system will have a list of active processing goals 38. 

30 Active goals are the processing goals for each process- 
ing module, which have yet to be accomplished by the 
system. At the start of processing this list will include the 
specific product or products requested by the customer, 
along with tiie translation of tiiese product goals into 

35 system and module goals. The customer profile 
includes botii factual and uncertain irrformation related 
to the customer. The factual data would include general 
items such as name, address names and ages of indi- 
viduals in tiie household, important dates, anniversa- 

40 ries, product preferences and purchase^* history. In 
addition factual data such as face recognition feature 
vectors of the immediate family eind extended family, 
voice, training sets, handwriting samples, would also be 
included in the customer profile database. Uncertain 

45 information would include processing goals recom- 
mended by processing modules from previous order 
processing, image understanding assertions about the 
contents of the image or groups of images which have 
not been verified or other unverified information or 

so, assertions. Information and data contained in the cus- 
tomer profile is updated with every order processed to 
reflect changes in order preferences, order history and 
update uncertain information. In order supply tiie sys- 
tem with the necessary amount of image understanding 

55 required to arrange the images into set in accordance 
with themes relating to predetermined criteria that must 
be provided for the system to have the capability to 
identify attributes within the images. For example, in a 
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preferred enr^odiment. if a customer requests a birth- 
day photocoUage the system will retrieve the default 
processing goals which indicate that the steps of col- 
lecting, sorting, culling, annotating, and composing will 
be involved in ^e processing of the requested product 5 
In addition the processing goals for each of the modules 
will reflect the default attributes necessary to process a 
birthday product. In tNs example the sorting module 
processing goals will include a sort by date goal and a 
sort by content goal. The sort by date processing goal is 10 
further refined to sort the images that occur on or near 
a target list of dates which are determined from a list of 
known birthdays retrieved from the customer profile Into 
the photocollage. 

[0018] In a preferred embodiment each processing is 
module performs processing on two distinct levels: 
objective and subjective. The objective processing 
deals with factual information about the images or 
groups of images or with data calculated deterministic 
algorithms. Examples of factual data include the size of 20 
the image, capture parameters, histograms, image 
transforms, etc. Commercially available software pro- 
grams such as Adobe Photoshop ® and Corel Draw ® 
process images using objective data. Subjective 
processing deals with information that is uncertain or 2S 
data Is the result of non-deterministic algorithms. Often 
subjective results carry with them a confidence fector 
which allows subsequent processing steps to interpret 
the results. Usually subjective results occur when 
attempting to determine abstract information about an 30 
image or group of images. Examples of tills type of the 
processing would be face detection, iace recognition, 
facial expression determination, location determination, 
assertions, interpretations, etc. Commercially a\^ilable 
software programs such as FacelT ®^ by Visionics Corp. as 
process images to associate faces in images with 
names and other information. Some processing mod- 
ules process only subjective information, others proc- 
ess only objective information and still, others process 
botii. 40 
[0019] As shown in FIG 3 the processing of a pho- 
tocollage is directed by the system using the active 
processing goals 38, 48, and 49. Because the process- 
ing goals are non-deterministic, vary by requested prod- 
uct and may be modified during the processing there 45 
exist a large number of possible processing sequences. 
The progression of processing applied to the collected 
image material is applied in sequential steps. Each step 
implements a single processing module that is intended 
to satisfy or partially satisfy an active processing goal, so 
At each step in tiie process any one of the available 
processing modules (culling 62. groining 64, enhancing 
66, annotating 68, associating 70, or composing 72) 
may be executed. At the conclusion of the processing of 
a particular step the active processing goals 38 that ss 
existed before the step processing are updated to 
reflect the changes in goals that resulted from the exe- 
cution of the processing step 52. These updated active 



processing goals 48 serve as the input processing goals 
for the subsequent processing step 53. Each of the 
available processing modules wOt be descn'bed. In addi- 
tion to the processing goals 38. 48, and 49. the process- 
ing modules at each processing step have access to 
four sources of infbrmatbn: pixel data 32 contained in 
the individual images, metadata 34 attached to images 
or groups of images, the original product or service 
order description 36 and, a customer profile 30 cdrrtain- 
ing information about the customer. This process can 
iterate as many times as necessary to complete the 
desired product objectives. 

[0020] The processing of the collection of digital 
image material is reviewed to remove unwanted ' 
images. This step is called culling. Culling is performed 
on images that are deemed unwanted due to poor qual- 
ity, or if there are several images that are similar in com- 
position and only one is desired. To accomplish this 
culling process Involves the computation of several 
image based metrics using the pixd data 32. These 
metrics fall into two basic categories: technical quality 
measures and abstract feature metrics. The technical 
quall^ measures would Involve calculations to assess 
the overall sharpness of tfie image, exposure quality, 
grain quality of tiie image. Algorithms used to calculate . 
technical quality measures are common in the art, 
examples would Include: "Estimation of Noise in 
Images: An Evaluation" by S. I. Olson in, 55 (4), 1993. 
pp. 319-323 and "Refined filtering of image noise using 
local statistics" by J. S. Lee in Computer Vision Graph- 
ics and Image Processing, 1 5. 1 981, pp..380-389. 
[0021] Images are then given an overall rating of 
technical quality based on the technical quality metrics. 
The overall image quality metric is then compared to a 
tiireshold associated with the processing goal. Images 
whose quality ratirig falls below the threshold' are 
flagged as unwanted. The specific threshold applied to 
a given image or set of images is parametrically deter- 
mined using tiie processing goal 38 and the customer 
profile 30 as inputs. In addition to the tietermlnaflon of 
specific objective image features, abstract subjective 
image features are. also calculated. These subjective 
measures of the image allow the module to cull 
unwanted or unnecessary Images. A typical exanple of 
images which require culling occurs when several 
images in tfie group are simllariy composed arxJ contain 
nearly identical content such as a group photo which is 
repeated several times. In order to choose tiie best of 
tills group of images the digital image is analyzed to 
determine a quality metric composed of both objective 
and subjective quality features. The basis for this sub- 
jective assessment would be tiie presence or absence 
of faces, the identification of the faces In the image, the 
number of faces present, if the eyes are open on each 
face, if the subject is smiling, or orientation of the face to 
the camera. Examples of algorithms used to deterrrv'ne 
subjective features of an image are desabed in Pro- 
ceedings of the IEEE Computer Society conference on 
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Computer Vision and Pattern Recognition, June 1997. 
These subjective features provide clues about the sub- 
jective quality of the image. The objective and subjec- 
tive features calculated for each image are combined 
and the images are ranked according to the relative 
importance of each feature analyzed to the processing 
goal 38 and which matches the stored customer prefer- 
ences 30. The ranking, because it is partially based 
upon subjective information also carries a probability 
weighting factor. The combination of the ranking and the 
probability weighting factor is used to assert a decision 
about the quality of the image. This quality assertion is 
retained with the image as metadata 34 and is passed 
to the next processing step, in the grouping step images 
are grouped according to criteria derived from the anal- 
ysis of the customer profile 30, the active processing 
goals 38 and the requested product or service 36. The 
goal of grouping images is to associate images that are 
a part of a common theme or story in the mind of the 
customer. Typical groupings derived from customer pro- 
files 30 would be to associate images according to the 
preferred organization scheme of the customer. The 
attributes of typical organization schemes would include 
organizing by events such as birthday party, vacation, 
holiday, graduation, organizing by time, and organizing 
by people. Customer requested products 36 could also 
dictate grouping parameters in accordance with the 
product design. Examples of these grouping schemes 
include grouping by location in a theme park, by com- 
mon life-cycle event such as graduation, by image con- 
tent type such as outdoor, indoor, nature, person, group, 
etc. A variety of software algorithms exist which are 
applicable to grouping. One class of software algo- 
rithms operates on metadata 34 associated with the 
image where the grouping process is similar to search- 
ing a media database. Several commercial examples of 
media databases that allow searching are Picture Net- 
work Incorporated, Publishers Depot (www.pubfish- 
ersdepotcom) . A second class of algorithms employed 
for this grouping process would include image process- 
ing algorithms which identify objects and feature sets 
within an image by way of the pixel information 32. 
These objects could represent typical cultural icons 
such as birthday cakes, Christmas trees, graduation 
caps, wedding dresses, etc. An example of such algo- 
rithms is reported by J. Edwards and H. Murase, 
"Appearance Matching of Occluded Objects Using 
Course-to-fine Adaptive Masks" in Proceedings of the 
IEEE Computer Society Conference on Corrputer 
Vision and Pattern Recognition. June 1997, pp. 533- 
546. The Product Order Information 36 can also be 
used in this grouping by simply stating the event or loca- 
tion or person on which to base the final product. The 
Processing Goals 38 can also be used as a grouping 
tool where the specific products offered mandate a spe- 
cific grouping. 

[0022] The annotation step seeks to generate 
annotation that can be composed on the story product 



or photocollage. Annotation is designed to provide con- 
text information about tiie image or group of images and 
assist the customer in communicating the story sur- 
rounding the images. A very common form of annota- 

5 tion is text associated with each image and with a group 
of images which explains the Vho, what, when, where, 
and why". Such context information is generally derived 
from metadata 34 generated in previous processing 
steps, user profiles 30 or, image understanding algo- 

10 rithms applied in the annotation module. "Who" informa- 
tion may be determined using face recognition 
algorithms applied to the pixel data 32 tuned by the 
training data contained In the customer profile 30. Using 
a commercial product such as "Facelt" by Visionics, a 

75 database of known faces can be retrieved from the cus- 
tomer profile 30 and used to guide the face recognition 
software. "What" information can be asserted by corre- 
lating date and time of the image capture available in 
the image metadata 34 witii key dates contained in the 

20 customer profile 30; In addition more "what" data can 
also be asserted by looking within the image 32 for spe- 
cific cultural icons, derived from object recognition algo- 
rithms, such as birthday cakes, Christmas trees or, 
graduation robes. "When" infonnation for annotation is 

25 easily determined using metadata 34 such as date and 
time recorded at the time of capture. "Where" informa- 
tion may be provided from the capture device integrated 
with GPS (Global Positioning System) technology and 
then added to the metadata 34 for the image or it can be 

30 guessed at using image understanding algorithms 
which correlate known location scenes with tiie image 
content from available images 32. "Why" information is 
very difficult to determine without some input from the 
image owner via the product order information 36. The 

35 annotation, once determined, can be rendered as text, 
graphics, images, video or sound and associated with 
tiie individual images or witii the groi^ of images in 
eitiier a traditional photocollage or digital product. 
[0023] The image enhancement module applies 

40 image processing to improve the image for jhe intended 
story based purpose. Two categories of image 
enhancement are considered for each image. Basic 
image enharicements are applied to all images, as nec- 
essary, would include: red-eye removal, brightness and 

45 contrast adjustment, color adjustment, and crop and 
zoom to improve composition. Image enhancements 
are applied directiy to tiie pixel data 32 and recorded in 
tiie image metadata 34 from tiie images. A second cat- 
egory of image enhancement is applied to the images in 

50 order to enhance their use within a story context to com- 
municate the emotion or to support the story theme as 
described in the processing goal 38. For instance, 
selected images from a wedding story would be sof- 
tened using a blur algorithm or the colors of an image 

55 could be made to match those found in a comic strip. 
The application of this type of enhancement processing 
is either specified in the product description 36 gener- 
ated from the customer profile 30. All of these opera- 
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tions are available in image editing software programs 
such as Adobe PhotoShcp. with the exception of red- 
eye removal and red-eye removal can be acconplished 
via the Kodak Imaging Workstation and other software 
such as Picture-It from Microsoft. 
[00241 Some product specifications will include a 
processing goal to associate external content that is rel- 
evant to the product and the image content This asso- 
ciated content provides additional context and interest 
in the final rendered product. External content takes 
several forms including weather reports, newspaper 
headlines, stock photographs, advertisements, histori- 
cal references, travel brochure copy, popular music 
clips, video segments, etc. In order to locate and find 
appropriate content for the image story / photocollage 
the metadata 34 attached to each image and image 
group and is interrogated to derive searchable topics or 
specific information located in the customer profile 30. 
These search topics would focus on the when, where, 
what aspects of the image story. Once a series of poten- 20 
tial search topics have been assembled they are format- 
ted into queries and searches are performed on a 
variety of databases. The results of these content que- 
ries is then refined by applying priority rules from the 
.product description database and customer prefer- 2s 
ences stored in the customer profile 30. These types of 
searches are common in information databases such as 
Yahoo® on the intemet (www.vahQo cnnr^) or from Dia- 
log corp. If there are items within the image that can be 
identified with specific events (such as a wedding or 30 
birthday) or if a person can be identified from imaging 
algorithms, the Pixel Information 32 is used to determin- 
ing the types of additional content that are added. The 
Processing Goals 38 dktete the forms and types of 
associated content that is added. The Product Order 55 
Infomiation 36 is also a source of information regarding 
the association of content by a speciEd request on the 
form (such as a theme park or wedding motiQ. 
[0025] The layout processing module places the 
image data, annotations, and associated content into an 40 
output ready format. This layout step must be repeated 
for each product that was requested by the customer 
and for each layout goal that was generated during the 
processing steps. The main task involved in the layout 
of a multi-page photocollage is the determination of 4s 
which images are to be placed in specific locations on 
specific pages. In addition the external content which 
has been associated with the individual images or 
groups of images must be composited onto the page. 
The layout process may be accomplished in numerous so 
ways. One means of performing this processing step is 
to employ a parametric photocollage description. This 
description specifies the general structure of the photo- 
collage but is not sufficient, of itself, to specify a final 
photocollage product The description file includes pho- ss 
tocollage features such as number of sections, size and 
format of individual pages, maximum and minimum 
number of pages allowable, maximum and nrBnimum 



number of customer images allowable per page, section 
descriptions, cover designs, location of allowable stock 
content, etc. By enplpying a photocollage description 
file a variety of tailored photocollage products may be 
5 designed incorporating features specific to the design 
goals. The layout step begins by reading the* photocol- 
lage description file to initialize a layout algorithm. The 
layout algorithm then worte page by page and section 
by section to apply specific layout mies guided by the 
10 values in the photocollage description fila A variety of 
page/section layout algorithms could be employed 
which embody different design philosophies for photo- 
collage products. The final photocollage product can be 
rendered on a variety of output media types which 
IS would include paper, fabric as a d^ital file or on any 
one of a number of digital media forms such as a CD. 
The Processing Gtoals 38 are used to determine the 
capabilities of the specific, devices being used. Cus- 
tomer Profiles 30 are used to determine color prefer- 
ences, layout preferences, and design considerations. 
Metadata 34 is used to determine the placement of 
assodated content with specific images. 
[0026] Fig. 4 shows the system diagram for Story 
Preparation Services. Input to the system comes from 
Personal Computers 90 (using albuming software such 
as Microsoft Picture ft, Family Base from Micro Dynam- 
ics or a myriad of others), an interactive Wosk. via the 
phone or over-the-counter order forms 84, a retail outlet 
82 with links to the system, or digitization services 80 
specializing in converting analog material Into digital 
information. The information needed to perform tiie 
story sen/ices is communicated via traditional means 
including mall, phone, modem, Internet or other on-line 
services. The components required by such as system 
include a digitization system for pictures, video arid 
audio 104, a storage facility 102, an operator interface 
96, algorithnrts and software 98 for the analysis of the 
data and to provide the necessary steps to complete the 
product, and an output delivery system 100 for printing 
or other media. . « 

[0027] Other aspects of the invention include: 

1. The method wherein the sorting step hjrther 
comprises categorizing according to one of the fol- 
lowing: (an, event; a person; or chronologically). 

2. The method wherein the predetermined criteria 
further comprises analyzing pixel data for quality 
and duplicate images. 

3. The method wherein the step of employing 
means respor^ive further comprises applying one 
of following (enhancements; associations; annota- 
tions; or layouts) to compose the photocollage. 

4. The method wherein the step of obtaining further 
comprises scanning the images to obtain the digital 
records of the images. 

5. The method wherein tiie step of obtaining the 
digital records further comprises receiving the dig- 
ital records of the images via communication 
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means. 

6. The method wherein the step of obtaining the 
digital records further conprises providing the dig- 
itaJ records of the images. 

7. The method wherein the step of obtaining further s 
comprises obtaining the plurality of images with 
identification information. 

8. The method wherein the step of obtaining further 
comprises the gathering of additional data with 
regard to the plurality of images. io 

9. The method wherein the additional data is meta- 
data selected from one of the following categories: 
(date; time; location; Advanced Photo Standard 
information on exposure, sound files, or camera 
identification number). 

10. The method wherein the step of gathering addi- 
tional data further comprises the collecting of cus- 
tomer order information. 

1 1 . The method wherein the gathering of additional 
data includes maintaining a customer profile perti- 20 
nentto creating photocollages. 

12. The method wherein the step of sorting the dig- 
ital records further comprises using the customer 
profile as at least a portion of the predetermined cri- 
teria. ^ 

13. The method wherein the step of analyzing the 
digital records further comprises analyzing the dig- 
ital records in accordance with customer profiles to 
determine the al least one attribute of the digital 
records. ^ 

14. The method wherein the step of analyzing fur- 
ther comprises analyzing at least one objective fea- 
ture of the digital records of the images to 
determine the at least one attribute. 

15. The method wherein the step of analyzing fur- 35 
ther conprises analyzing at least one subjective 
feature of the digital records to determine the at 
least one attribute. 

16. The method wherein the step of automatically 
sorting the digital records further comprises identi- 40 
tying and collecting together duplicates of the digital 
records. 

17. The method wherein the step of employing 
computational means for sorting the digital reconds 
further comprises identifying, and collecting 45 
together digital records of the images that have a 
subjective quality rating below a predetermined 
threshold. 

18. The method wherein the step of employing 
computational means for sorting the digital, reconds so 
further comprises identifying and collecting 
togetiier digital records of the images tiiat have a 
objective quality rating below a predetermined 
threshold. 

19. The method wherein the step of enploying - ss 
computational means for sorting said digital records 

in accordance with the at least one attribute further 
comprises applying one of the following attributes: 



(event; location: chronology; personal information; 
objective criteria; or subjective criteria). 

20. The method wherein tiie step of analyzing tiie 
digital records further comprises automatic deter- 
mination of annotations for the digital records. 

21 . The method wherein the step of applying further 
coniprises enhancing tiie digital records. 

22. The method wherein the step of appiy*ng further 
comprises placing and assodating annotations with 
the digital records in the photocollage. 

23. The method wherein the step of applying further 
comprises determining additional content to be 
included in the photocollage. 

24. The method wherein the step of applying further 
coniprises placing adcfitional content witiiin the 
photocollage. 

25. The method wherein the step of applying further 
comprises placing the digital records into tiie photo- 
collage according to the predetermined criteria. 

26. The system wherein the device for obtaining a 
digital record further comprises connputational 
means for uniquely identifying the digital records. 

27. The system wherein ttie predetermined criteria 
further comprises a data type that is selected from 
one of the following: (pixel data; metadata; product 
order Information; processing goal information; or 
customer profile). 

28. The system wherein the means for sorting fur- 
tiier comprises applying the at least one predeter- 
mined criteria to sort in terms of one of the 
following: (culling or grouping). 

29. The system wherein the means for sorting fur- 
ther comprises categorizing to one of tiie following: 
(an event; location; chronology: personal informa- 
tion; objective criteria; or subjective criteria). 

30. The system wherein the means for sorting cate- 
gorizes by culling the digital records employing an 
analysis of pixel data for quality and duplicate 
images. 

31 . The system wherein tiie means fbr^rting com- 
prises categorizing mean for grouping according to 
image related information. 

32. The system wherein the image related infbrina- 
tion used in the means for sorting is image related 
information relating to either: (an event; location; 
chronology; personal information; objective criteria; 
or subjective criteria). 

33. The system wherein the means for composing 
furtiier comprises means responsive to the means 
for sorting for employing one of the following: 
(metadata; pixel information; processing goals; 
customer information or product order information). 

34. The system wherein tiie means responsive fur- 
ther comprises applying one of following (enhance- 
ments; associations; annotations; or layouts) to 
conpose the photocollage. 

35. The system wherein the device for obtaining a 
digital record further comprises means for scanning 
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the images to obtain the digital records of the 

images. 

36. The system wherein the device for obtaining a ' 
digital record further comprises means receving 
the digital records of the images via communication s 
means. 

37. The system wherein the device for otstaining a 
digital record further comprises provkiing the digital 
records of the images. 

38. The system wherein the device for obtaining a io 
digital record of the plurality of images further conrv 
prises means for creating a digital Identification for 
each of the digital records from identification infor- 
mation provided within each respective of the plu- 
rality of images. 'IS 

39. A system for produdng a photocollage from a 
plurality of images, comprising: 

a) con^utational means operatively connected 

to a device for obtaining a digital record for 20 
each of the plurality of images, each of the dig- 
ital records having a unique identifier and stor- 
ing the digital records in a database; 

b) means for analyzing the digital records 
within the computational means to obtain infbr- 2S 
mation with respect to at least one attribute of 

the digital records; 

c) means for sorting the digital records in 
accordance with the at least one attrbute 
within the computational means; and - 30 

d) means for applying the at least one attribute 
according to a predetermined criteria using the 
computational means to compose a photocol- 



In accordance with the at least one of the following 
attributes: (event; location; chronology; personal 
irrformation; objective criteria; or subjective criteria). 

46. The system wherein the means for analyzing 
the digital records further comprises means for 
determination of annotations associated with the 
digital records. 

47. The system further comprising withifi the 
means for applying means for enhancing of tffe dig- 
ital records used in creating the photocollage. 

48. The system wherein the means for applying fur- 
ther comprises means for placing and associating 
annotations with the digital records in the photocol- 
lage. 

49. The system wherein the means for applying fur- 
ther comprises means for pladng the digital records 
in the photocollage according to the predetermined 
criteria. . '■ 

50. The system further comprising means for gen- 
erating a visually discernable version of the photo- 
collage. 

51 . The system further comprising means for creat- 
ing a digital recording of the photocollage. 

52. The system further comprising an operator 
interface to communicate with the computational 
means. 



Claims 



35 

40. The system wherein the rheans for analyzing 
said plurality of images further conprises analyzing 
the plurality of images in accordance with customer 
profiles to determine at least one said attribute of 
said plurality of images. 40 

41 . The system wherein the means for analyzing 
further comprises analyzing at least one objective 
feature of the digital records of the images to deter- 
mine the at least one attribute. 

42. The system wherein the means of analyzing fur- 45 
ther conprises analyzing at least one subjective 
feature of the digital records to determine the at 

least one attribute. 

43. The method wherein the means for sorting the 
digital records further comprises means for identify- so 
ing and collecting together duplicates of the digital 
records. • 

44. The system wherein the means for sorting the 
digital records further comprises means for identify- 
ing and collecting together digital records of the ss 
images that have unacceptable quality. 

45. The system wherein the means for sorting the 
digital records further comprises means for sorting 



1. A method of producing a photocollage from a plu- 
rality of images, comprising the steps of: 

a) obtaining a digital record for each of the plu- 
rality of images, each of the digital records hav- 
ing a unique identifier and storing the digital 
records in a database; 

b) automatically sorting the digital records 
using at least one date type to categorize each 
of the digital records according at least one 
predetermined aiteria; « 

c) employing means responsive to the sorting 
step to compose a photocollage from the digital 
records 

2. The method according to claim 1 wherein the step 
of automatically sorting further conhprises as the at 
least one data type, a data input that is selected 
from one of the following: (pixel data; metadata; 
product order information; processing goal Informa- 
tion; or customer profile). 

3. The method according to claim 1 wherein the sort- 
ing step further comprises employing one of the fol- 
lowing: (culling or grouping). 

4. The method of claim 1 wherein the categorization is 
determined by grouping according to image related 
information. 
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5. The method of claim 4 wherein the image related 
information used in the sorting step is selected from 
either: (an event; location; chronology; personal 
information; objective criteria; or subjective criteria). 

5 

6. The method of claim 1 wherein the step of obtaining 
further con^rises obtaining with the digital record 
data relating to one of the following: (metadata; 
customer profile information; or product order inffor- 
nr^tion). 

7. The method of daim 1 wherein the step of employ- 
ing means responsive further comprises employing 
one of the following: (metadata; pixel information; 
processing goals; customer information or product is 
order Information). 

8. A photocollage product comprising the steps of: 

a) obtaining a digital record for each of the plu- 20 
rality of images, each of the digital records hav- 
ing a unique identifier and storing the digital 
records in a database; 

b) analyzing the digital records to obtain infor- 
mation with respect to at least one attribute of 25 
the digital records; 

c) employing computational means for sorting 
said digital records in accordance with the at 
least one attribute; and 

d) applying the at least one attribute according 30 
to a predetermined criteria using the computa- 
tional means to compose a photocollage. 

9. A system for producing a photocollage from a plu- 
rality of images. 3S 
the system comprising: 

a) a device for obtaining a digital record for 
each of a plurality of images; 

b) means for automatically sorting the digital 4o 
records in accordance witii at least one prede- 
termined criteria; and 

c) means for cornposing a photocoDage of the 
digital records in accordance with tiie predeter- 
mined criteria. ^ 



so 



55 



10 



EP1 004 967A1 



8 





1 1 





\ 




--12 



MEMORY CARD 
I 




ALBUMS 



-10 



EXPOSED FILM 




6 
/ 




STORY 
PREPARATION 
SERVICES 




OTHER 
KEEPSAKES 



PRINTS 



FIG. 1 



11 



EP 1 004 967 A1 



8 

1 




14 



PROCESSING 
FACILITY 



DIGITIZATION 

SERVICES 



16 



22 



PROCESS 



24 



1 




\ 



) 



40' 



DEFAULT 
PRODUCT 
PROCESSING 
DESCRIPTION 



18 



SCAN 



20 



32' 
34' 

36' 
30' 



PIXEL INFORMATION 



METADATA 



ORDER INFORMATION 



CUSTOMER PROFILE 




PROCESSING 

GOAL 
GENERATION 



38' 



PROCESSING 
GOAL 



y 



FIG. 2 



12 



EP1 004 967A1 



PROCESSING 
GOAL 



'38 



52 
1- 



62- 



CULLING 



6"^-^^ GROUPING 



6^ ENHANCING 
68- 



ANNOTATING 



^^ASSOCIATING 
72 



COMPOSING 



REVISION 1 
PROCESSING 
GOALS 




62- 
64-v. 

66^ 

68^ 

72- 



CULLING 



GROUPING 



ENHANCING 



ANNOTATING 



ASSOCIATING 



COMPOSING 



REVISION 2 
PROCESSING 
GOALS 



54 



32- 

34' 
36' 
30' 



PIXEL 

INFORMATION 



METADATA 



ORDER , 
INFORMATION 



CUSTOMER 
PROFILE 



32 

34' 
36' 
30' 



PIXEL 
INFORMATION 



METADATA 



ORDER 
INFORMATION 



CUSTOMER 
PROFILE 



32- 

3.4' 
36' 
30' 



PIXEL"- 
INFORMATION 



METADATA 



ORDER 
INFORMATION 



CUSTOMER 
PRORLE 







62-v. 


CULLING 1 


64-v. 


GROUPING 1 


66-v. 


ENHANCING | 


68^ 


ANNOTATING | 


70^ 


ASSOCIATING 1 


72-^ 


COMPOSING 1 









FIG. 3 



13 



EP1 004 967A1 



DIGIT- 
IZATION 
S ERVICES 

T" 

80 



84^ 

RETAIL 
OUTLET 



T 

82 




WRITTEN 
ORDERS PERSONAL 
ORDERS 





1 



DIGITIZATION 
SYSTEMS 



y 

104 



STORAGE 




1 



96' 



OPERATOR 
INTERFACE 



100' 



102 




ALGORITHMS 
AND DESIGN 
SOFTWARE 



OUTPUT 
METHODS 



FIG. 4 



14 



EP1 004 967A1 



European Patent 
Office 



EUROPEAN SEARCH REPORT 



AppHe«t)on Number 

EP 99 20 3782 



I Category 



DOCUMENTS CONSIDERED TO BE RELEVANT 



Citalkin of document with tncfication, where appropriate, 
of relevant passages * 



us 5 761 655 A (HOFFMAN MICHAEL T) 
2 June 1998 (1998-06-O2J 

* abstract * 

* column 4, line 8 - line 34 * 

DAS M ET AL: "FOCUS: SEARCHING FOR 
MULTI-COLORED OBJECTS IN A DIVERSE IMAGE 
DATABASE" 

PROCEEDINGS OF THE IEEE COMPUTER SOCIETY 
CONFERENCE ON COMPUTER VISION AND PAHERN 
RECOGNITION.US.LOS ALAMITOS. IEEE COMP. 
SOC. PRESS, 

vol. CONF. 16, page 756-761 XP00Q776573 
ISBN: 0-7803-4236-4 

* abstract * . 

US 5 751 286 A (BARBER RONALD JASON ET 
AL) 12 May 1998 (1998-05-12) 

* coluran 13. line 44 - column 17. line 58: 
figures 12-14 * 



ET AL) 



US 5 576 759 A (KAWAMURA KOICHIRO 
19 November 1996 (1996-11-19) 
* colunn 2. line 38 - coluran 3, line 4; 
figures 9A,9B * 



Th» present aaaieh report has been drawn up for aO daims 



Relevant 
todaim 



1-9 



1.2.8 



1.2 



CLASS0=ICAT1ON OF THE 
APPUCATION (lnta.7) 



G06F17/30 



TECHNICAL FIELDS 
SEARCHED (lnta.7) . 



G06F 



BERLIN 

CATEGORY OF CfTED OOCUMEl^ 



of eon^plalion flf Iha 

6 December 1999 



Deane» E 



X : paftioutariy mlflvont 9 takett ek»ne 

y : paiticulaffy mtavant V oonAiinad w»i anoHMr 

dooument or th* MHM eotoQory 
A : tMhnolooioatbaekBfQund 

O : nof^-wnttsn ' 

P : Intormedii 



T : th«c^ Of pfifiopte undwfyihg Ihe bwentJoii 
E : ftari-«r patent dooumant, but pulsisliMl on. or 

«ft»f tlM mff>g date 
D : doeumont oftod in 0w Mplieatbfl 
L : daotimMtekod for othor lessons 



Ainwrnborafihei 



» patent fsmiy, eorrtapoi^g 



15 



EP1 004 967A1 



ANNEX TO THE EUROPEAN SEARCH REPORT 

ON EUROPEAN PATENT APPLICATION NO. EP 99 20 3782 



TWi annex Mb (he patent family memben relating to the patent dooumanti oiled in the abOOTMnentioned Eumpewi Mwrcb report. 

Hie membera are ai contained in the EurepoinPatartlOISoeEDP Be on . . , ^ < «„ 

ThJ ESSp^^tSl Office i> in no way Ibrtheie partioulare which are merely given lot the purpose dinfonnahon. 

Q6-12-1999 



Ptf ant dooumant 
died in aeareh report 


PuMeation 
data 


Patent family 
menit>ei<a) 


1 Publication 
date 


US 5761655 


A 


02-Q6-1998 


NONE 




US. 5751286 


A 


12-D5-1998 


US 5579471 A 


26-11-1996 


US 5576759 


A 


19-11-1996 


JP 6178261 A 


24-66-1994 



s " ■ - 

i 

i • • : 

& For more detaBs about this annex : see Official Journal of the European Patent Office, No. 12/B2 



16 



